The MAVLink is a lightweight communication protocol between Unmanned Aerial Vehicle (UAV) and Ground Control Station (GCS). It defines a set of mutual bi-directional messages between UAV and GCS, including UAV states and GCS control commands. However, the MAVLink protocol lacks sufficient security mechanisms, and there are security vulnerabilities that may cause serious threats and hidden dangers. To resolve these problems, a security communication scheme for the UAV system based on the MAVLink protocol was proposed. First, the connection requests were broadcasted by the UAV constantly and alternately; then the public key was sent to the UAV by the GSC, and the DH algorithm was used by both sides to negotiate a shared key, and the AES algorithm was used to encrypt the communication on MAVLink message packages, achieving identity authentication. If the UAV did not receive the public key sent by the GCS within the specified time or a decryption error on MAVLink message package happened, the UAV would actively disconnect and update a new public key to rebroadcast the connection request. In addition, concerning the security problem of the UAV system being maliciously tampered with, the system firmware was self-checked during booting. Finally, based on the formal verification platform UPPAAL, it has been proved that the proposed scheme has the security properties of liveness, connectability and connection uniqueness. Results of the communication process between UAV PX4 1.6.0 and GCS QgroundControl 3.5.0 show that the proposed secure communication scheme of UAV system can prevent malicious eavesdropping, message tampering, man in the middle attack and other malicious attacks in the communication process between UAV and GCS, and solve the security vulnerabilities of MAVLink protocol well with little effect on UAV performance.